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16.  Abstract 

The  Internet  affords  an  increasingly  capable,  economical,  and  popular  vehicle  for  preflight  weather  self-briefing. 
This  research  constitutes  the  first  known  experimental  investigation  of  how  modern  Web-based  weather  products 
are  actually  being  used  by  general  aviation  (GA)  pilots. 

A  data-gathering  emulation  of  the  National  Weather  Service  Web  site  http://aviationweather.gov  was  written  for 
use  on  a  personal  computer.  Eighteen  Web  pages  were  created  to  display  weather  information,  each  page 
emulating  a  similar  category  of  product  shown  by  aviationweather.gov.  This  formed  the  primary  weather  briefing 
site  for  two  similar,  challenging  weather  scenarios  subsequently  flown  by  50  GA  pilots  in  simulated  cross-country 
VFR  flight. 

Two  dependent  variables  were  measured — page  views  (which  pages  each  pilot  viewed)  and  pageview  duration 
(how  long  each  page  was  viewed).  Total  briefing  time  was  then  calculated  for  each  pilot  by  summing  pageview 
durations.  A  groupwise  “dwell”  index  was  also  derived  for  each  Web  page  by  multiplying  the  group’s  page  views 
times  its  median  pageview  durations. 

A  number  of  specific  findings  emerged  from  the  data  analysis.  First,  given  a  specific  flight  mission,  we  found 
that,  as  a  group,  GA  pilots  appeared  significantly  consistent  over  time  in  the  kinds  of  information  they  sought  out. 
However,  the  amount  of  time  they  spent  examining  that  information  seemed  to  vary  over  time,  even  given  similar 
flight  situations. 

Second,  pilots  seem  to  have  favorite  information  sources  (e.g.,  graphical  prog  charts,  TAFs,  NEXRAD,  and 
satellite  cloud  images).  We  could  expect  those  favorites  to  shift  slightly,  depending  on  the  flight  mission. 

Third,  Web  page  design  is  potentially  important.  For  example,  pages  not  accessible  from  the  top-level  menu  may 
be  ignored  or  overlooked. 

These  findings  are  important  to  weather  information  providers  because  the  ability  to  display  certain  types  of 
information — and  certain  ways  of  displaying  it — are  sure  to  give  the  knowledgeable  provider  an  advantage  over 
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Figure  1.  The  basic  data  path  of  weather  information  (ASOS  and  NEXRAD  are 
explained  below). 


1807 

1891 

1946 


Coast  and 

Geodetic 

Survey 

Weather 

Bureau 

Central  Radio 

Propagation 

Laboratory 


1965 

Environmental 
Science  Ser¬ 
vices  Adminis¬ 
tration  (ESSA) 


► 


1970 

National  Oceanic 
and  Atmospheric 
Agency  (NOAA) 


► 


1971 

National 

Weather 

Service 

(NWS) 


Figure  2.  Main  predecessors  of  today’s  U.S.  National  Weather  Service. 


INTRODUCTION 

Purpose  of  This  Research 

During  2008,  FAA  Civil  Aerospace  Medical  Insti¬ 
tute  (CAMI)  researchers  were  tasked  by  the  FAA  Flight 
Standards  division  (AFS-810)  to  explore  several  issues  in 
general  aviation  (GA),  including  how  modern  Internet- 
based  weather  products  are  used  during  preflight  briefing 
(Knecht,  Ball,  &  2010a,  b).  This  began  a  human  factors 
study  of  what  promises  to  be  the  future  of  GA  preflight 
weather  briefing — self-briefing  by  pilots  using  Internet- 
based  tools.  The  creation  of  the  software  used  to  test  that 
self-briefing,  the  methodology,  and  the  results  of  statistical 
tests  form  the  subject  of  the  current  report. 

The  idea  that  Web-based  self-briefing  will  become  a 
major  element  of  flight  preparation  requires  a  convinc¬ 
ing  argument.  We  begin  by  observing  that  the  necessary 
informational  infrastructure  is  here  already  and  can  be 
expected  to  expand  and  improve.  Let  us  briefly  review 
the  basics  of  that  infrastructure. 


The  Basic  Data  Path 

Weather  information  follows  a  basic  data  path  from 
Nature  to  end-user.  Figure  1  shows  that  path,  starting 
with  adverse  weather  and  ending  with  a  pilot,  either  on 
the  ground  or  airborne.  The  report  you  are  now  reading 
will  focus  on  the  “Internet  pathway”  for  preflight  briefing 
(as  opposed  to  the  in-flight  pathway  for  updates). 

A  Brief  History  of  Preflight  Weather  Briefing 

In  the  early  days  of  flight,  “weather  briefing”  consisted 
of  stepping  outside  to  see  what  was  going  on,  as  best  one 
could.  The  world  has  changed  considerably  since  then. 
There  is  now  extensive  governmental  infrastructure 
dedicated  to  gathering,  processing,  and  making  public  a 
wide  variety  of  meteorological  information. 

In  the  U.S.,  this  process  formally  began  in  1807  with 
President  Thomas  Jefferson’s  “Survey  of  the  Coast”  proj¬ 
ect.  Figure  2  shows  a  highly  abbreviated  history  of  U.S. 
weather  agencies  (condensed  from  Shea,  2010). 

Currently,  the  National  Oceanic  and  Atmospheric 
Agency  (NOAA)  serves  as  the  parent  organization  of  the 
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COMPLETED  WSR-88D  INSTALLATIONS 
WITHIN  THE  CONTIGUOUS  U.S. 
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Figure  3.  a)  an  ASOS  station;  b)  U.S.  ASOS  coverage;  c)  NEXRAD  coverage;  d)  AWIPS  workstation;  e)  NOAA 
IBM  Power6  P575  supercomputer.  Photo  credits:  a-c,  e  (NOAA),  d  is  public  domain  per  author,  Jonathan  Lamb. 


National  Weather  Service  (NWS),  which  provides  U.S. 
aviation  most  of  its  weather  information. 

Today,  NOAA  and  the  NWS  provide  nearly  all  the 
raw  weather  data  used  in  U.S.  aviation.  We  typically  see 
these  raw  data  repackaged  as  mid-level  weather  informa¬ 
tion  products ,  for  example: 


•  AIRMET . Area  Meteorological  Forecast 

•  FA . Aviation  area  18-h  forecast 

•  FD . Winds  and  temperatures  aloft 

•  METAR . Meteorological  Aviation  Report 

•  PIREPS . Pilot  reports1 

•  SIGMET Significant  Meteorological  Forecast 

•  TAF . Terminal  Aerodrome  Forecast 


These  products  are  then  reorganized  and  bundled  by 
weather  information  providers  into  more  user-friendly 
systems  such  as  the  Direct  User  Access  Terminal  Service 
(DUATS),  commercial  products  by  companies  such  as 
XM  and  WSI,  and — of  course — the  Automated  Flight 
Service  Station  (AFSS). 

One  of  the  premier  weather  providers  emerging  today 
is  NWS5  www. aviationweather.gov.2 *  This  service  is  free 
to  the  public  and  is  highly  informative  and  useful.  For 

1  PIREPS  are  currently  collected  and  distributed  by  local  Flight  Service 
Stations,  rather  than  by  NWS. 

2An  alternate  name  for  this  service  is  the  Aviation  Digital  Data 

Service  (ADDS). 


these  reasons,  emulating  and  testing  aviationweather.gov 
became  the  focus  of  the  current  study. 

The  goal  of  all  these  information  providers  is  to  con¬ 
vert  what  would  otherwise  be  hopelessly  complex  masses 
of  raw  data  into  simpler,  more  meaningful  graphical  or 
textual  forms  that  help  pilots,  airline  dispatchers,  and 
air  traffic  controllers  make  logical,  timely,  safe  decisions 
about  the  movement  of  aircraft. 

NWS’  Primary  Technological  Systems 

Beginning  in  the  1980s,  the  NWS  itself  was  modern¬ 
ized  with  the  following  critical  technological  infrastructure 
(Figure  3  illustrates): 

1 .  ASOS:  (Figs.  3a,  b) .  Automated  Surface  Observation 
Systems  are  a  network  of  more  than  800  modular, 
automated  ground-based  sensor  units  collecting  wind 
speeds/directions,  temperature,  barometric  pressure, 
humidity,  precipitation,  and  visibility  data.  These 
were  deployed  in  the  early  1980s. 

2.  NEXRAD:  (Fig.  3c).  The  Next-Generation  Weather 
Radar  system  is  a  ground-based  network  of  1 59  Dop¬ 
pler  weather  radars.  Doppler  radar  is  quite  sensitive 
to  high-frequency  radar  phase  shifts  induced  by  hori- 
zontal movement of “hydro-meteors”  (e.g.,  rain  drops 
or  insects)  driven  by  wind).  This  enables  NEXRAD 
to  detect,  not  only  precipitation  and  speed\direction 
of  moving  storm  systems,  but  also  wind  flow  fields, 
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turbulence,  wind  shear,  and  micro-bursts.  These 
radars  were  first  deployed  in  1990. 

3.  AWIPS:  (Fig.  3d).  The  Advanced  Weather  Interac¬ 
tive  Processing  System  processes  data  from  satellites, 
surface  radars,  and  surface  observation  stations.  It 
integrates  and  displays  data  in  ways  useful  to  model¬ 
ers  and  forecasters  (e.g.,  superimposing  barometric 
pressure  isobars  from  surface  observation  stations  on 
top  of  known  storm  cells  provided  by  NEXRAD). 
AWIPS  was  deployed  in  the  late  1990s, 

4.  Supercomputing:  (Fig.  3e).  National  Oceanic  and 
Atmospheric  Administration  (NOAA)  supercom¬ 
puters  use  multiple  central  processing  units  (CPUs), 
operating  simultaneously  on  vast  amounts  of  sensor 
data,  to  rapidly  process  computationally  complex 
forecasts  and  weather  models  (NOAA,  2009).  The 
latest  NOAA  machines  now  utilize  up  to  5000  CPUs 
and  provide  sustained  computing  speeds  up  to  70 
teraflops.3 

The  net  result  of  modernization  has  been  to  afford 
faster,  more  extensive,  more  reliable,  and  more publically 
accessible  weather  information  than  ever  before.  Most  of 
this  accessibility  has  been  due  to  the  rise  of  the  Internet. 

The  Rise  of  the  Internet 

What  we  today  know  as  the  Internet  began  in  the 
mid-1960s  with  the  experimental  ARPANET  project  of 
what  is  now  the  U.S.  Defense  Advanced  Research  Proj¬ 
ects  Agency  (DARPA,  Waldrop,  2010).  Expanded  in  the 
mid-1980s  by  U.S.  National  Science  Foundation  (NSF) 
funding,  commercialization  of  the  Internet  began  in  the 
late  1980s.  Exponential  growth  quickly  followed.  As  of 
February  2010,  estimated  U.S.  Internet  usage  stands  at 
over  234  million  individuals  (Nielsenwire,  2010) — rep¬ 
resenting  about  75%  of  the  national  population  (U.S. 
Census  Bureau,  2009). 

The  point  of  this  history  is  to  demonstrate  that  the 
Internet  is  simultaneously  widely  available  and  popular — 
two  vital  preconditions  for  the  spread  of  an  information 
technology  such  as  preflight  weather  self-briefing.  In  fact, 
availability  and  popularity  are  now  so  extensive  that  the 
significance  of  the  “Internet  Revolution”  is  widely  likened 
to  Gutenberg’s  development  of  the  printing  press. 

Central  to  the  Internets  potential  growth  is  a  third 
precondition  involving  functionality,  namely  the  rapid, 
inexpensive  transmission  of  large  amounts  of  information. 
This  depends  on  average  transmission  speed. 

3  Teraflops  is  a  measure  of  computer  speed,  representing  “one  trillion 
floating-point  operations  per  second.”  A  floating-point  number  is  an 
integer  followed  by  a  decimal  (e.g.,  34.997),  typically  represented  in 
hardware  by  a  64-bit  number  (footnote  4  describes  bits).  A  flop  itself 
is  an  arithmetic  operation,  for  instance,  addition  or  multiplication. 


Figure  4.  Internet  transmission  rate  for 
Nielsen’s  computer  (used  by  permission  of 
author).  Note  that  the  x-axis  is  linear,  the  y- 
axis,  base-10  logarithmic.  The  linear 
regression  line  therefore  represents 
exponential  growth. 

Average  transmission  speed  (or  bandwidth)  itself  depends 
upon  two  essential  technologies:  a)  computers  to  generate 
and  receive  information,  and  b)  a  network  of  cables,  rout¬ 
ers,  transmission  lines  or  towers,  and  satellites  to  rapidly 
move  that  information  from  origin  to  destination.  Various 
formal  principles  describe  how  computational  speed  and  bit4 5 
transfer  rates  across  the  Internet  are  influenced,  including: 

1  Moores  Law  states  that  high-end  computers  will 
theoretically  double  in  speed  about  every  1 8  months 
(Moore,  1965). 3 

2  Nielsens  Law  states  that  high-end  users  Internet  con¬ 
nection  speeds  will  theoretically  double  about  every  2 1 
months  (Nielsen,  2010).  Figure  4  illustrates  Nielsen’s6 
own  personal  Internet  connection  speeds  increasing 
exponentially  over  time. 

3  Bloat  describes  an  opposing  trend — that  computer  code 
will  get  bigger,  less  efficient,  and  significantly  slower 
with  time  (Kennedy,  2008). 

4  Bulk-up7  describes  a  second  opposing  trend — that 
computer  applications  will  become  increasingly 
capable,  at  the  expense  of  becoming  more  complex 
and  computationally  intensive. 

4  A  bit  is  the  smallest  piece  of  transmittable  information,  representing 
either  a  0  or  1  in  base-2  (binary)  numbers.  In  actual  digital  transmission, 
a  0  is  typically  sent  as  a  low-strength  energy  pulse,  a  1  as  a  significantly 
higher-strength  pulse. 

5  The  original  idea  stated  that  the  number  of  transistors  on  an  integrated 
circuit  would  rapidly  double — from  which  we  can  easily  infer  that 
processing  speed  will  increase. 

6The  Nielsen  of  Nielsens  Law. 

7Bulk-up  is  a  term  coined  for  this  report.  It  is  meant  to  highlight  the 
difference  between  how  a  bodybuilder  consciously  develops  muscle 
(“bulks  up”)  versus  how  most  of  us  unconsciously  simply  put  on  fat 
as  we  age  (“bloat”).  However,  this  basic  comparison  must  certainly 
have  already  been  described  somewhere;  therefore,  no  credit  is  claimed 
for  the  underlying  idea. 
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Despite  bloat,  the  net  effect  of  all  these  influences  has 
been  a  rapid  net  increase  over  time  in  how  much  data 
the  average  computer  can  process  and  rapidly  display. 
Transmission  and  processing  speeds  have  finally  grown 
to  the  point  where  enormously  sophisticated  graphical 
images  can  be  transmitted  to  home  users  in  seconds  at 
affordable  prices. 

Net  result.  The  net  result  of  all  these  factors  is  the 
realization  that  we  finally  have  the  ability  to  generate 
and  deliver  large  amounts  of  relatively  low-cost,  high- 
quality  weather  information.  A  “critical  mass”  has  been 
achieved  in  terms  of  pilots  being  able  to  affordably  see 
the  same  kinds  of  high-level  information  that  professional 
weather  forecasters  see. 

The  Internet’s  Impact  on  Preflight  Weather  Briefing 

Internet  preflight  weather  briefing  seems  to  be  becom¬ 
ing  quite  popular,  particularly  among  younger  pilots.  A 
recent  study  by  Knecht  (2008,  Table  1)  shows  NOAA/ 
NWS  Internet  briefing  ranked  #2  in  relative  overall  value 
(“Rank”)  and  perceived  informational  value  (“Perceived 
value”)  by  230  GA  pilots  from  five  states  (CA,  OK,  ND, 
IL,  FL).  Median  pilot  age  was  23  (range  18-78),  median 
flight  experience,  245  h  (range  15-18,000). 

Considering  everything  just  previously  listed,  we  can 
assume  that  we  are  poised  at  a  tipping  point.  Internet 
preflight  weather  self-briefing  is  already  popular,  and  the 
supporting  technology  is  becoming  ever- faster  and  more 
powerful.  Therefore,  this  popularity  is  almost  certainly 
going  to  increase,  making  scientific  study  of  this  technol¬ 
ogy  a  timely  topic  for  human  factors  research. 

METHOD 

Background 

The  current  study  was  an  exploratory  project  with  two 
main  objectives:  a)  to  begin  directly  investigating  how 
GA  pilots  interact  with  a  PC  “Internet-based  weather 


information  provider,”  and  b)  to  start  learning  what 
kind  of  experimental  equipment  and  methodology  will 
be  needed  to  collect  accurate,  meaningful,  scientific  data. 

This  experiment  took  place  as  part  of  a  two-part  lon¬ 
gitudinal  study  conducted  at  the  FAAs  Civil  Aerospace 
Medical  Institute  (CAMI)  in  Oklahoma  City.  Phase  1 
examined  data  collected  from  January-July,  2008.  Phase 
2  data  were  collected  July-September,  2008.  Briefly,  50 
GA  pilot  volunteers  were  selected  to  participate  in  Phase 
1  with  informed  consent.  Six  were  unable  to  participate 
in  Phase  2,  dropping  the  final  N  to  44.  Half  the  pilots 
were  instrument-rated,  half  were  private  pilots.  Half  were 
“local,”  defined  as  currently  residing  in  Oklahoma.  The 
other  half  were  transported  to  CAMI  by  commercial 
carrier  from  a  wide  variety  of  locations  around  the  U.S. 

Apparatus  and  Procedure 

Flight  simulator,  mission,  and  terrain.  As  usual,  airframe , 
flight  mission,  and  terrain  formed  the  context  for  the  weather 
briefing.  The  airframe  was  the  CAMI  Advanced  General 
Aviation  Research  Simulator  (AGARS),  configured  as  a 
Piper  Malibu,  a  high-performance,  propeller-driven  small 
aircraft.  The  mission  was  instructed  to  be  an  east-to-west, 
90-minute,  visual-flight-rules  (VFR)  pleasure  flight  from 
Amarillo,  TX  (AMA)  to  Albuquerque,  NM  (ABQ). 

Terrain  and  weather  were  used  to  dynamically  increase 
pilot  workload.  Terrain  gradually  rose  during  the  first 
two-thirds  of  the  flight,  followed  by  a  dramatic  elevation 
change  during  the  last  third.  Meanwhile,  a  continuous 
layer  of  clouds  also  rose,  but  it  rose  less  gradually  than 
terrain.  Figure  5  shows  2-  and  3-D  views  of  one  pilots 
actual  flight  profile. 

The  net  result  of  the  situation  was  to  “squeeze”  pilots 
between  cloud  bases  and  terrain,  making  for  an  increasingly 
difficult  flight.  Adding  to  the  difficulty  were  deteriorating 
VFR  weather  conditions.  Visibility  began  at  8  nautical 
miles  (nm)  and  gradually  decreased  to  5  nm  approximately 
two- thirds  of  the  way  along  the  route. 


Table  1.  Normalized  ranks,  values,  frequency  of  use,  and  time  spent  using  weather  information 
providers. 


Provider 

FSS  (standard  briefing) 

Format 

telephone 

Rank 

0-1 

1.0 

Perceived  Value 
0-1 

1.0 

Used  on  % 
of  flights 

61.5 

Min  spent 
when  used 
9.1 

Est.  ave. 
min  spent 

5.6 

Public  NWS  or  NOAA  site 

Internet 

0.7 

0.8 

49.8 

13.9 

6.9 

DU  ATS 

Internet 

0.7 

0.7 

34.0 

8.9 

3.0 

Commercial  vendor 

Internet 

0.4 
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8.9 

1.5 

0.1 
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Other  sources 
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0.0 

0.0 
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Total  estimated  length  of  wx  brief  per  flight  19.8 
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Figure  5.  a)  2-D  flight  profile.  The  x-axis  shows  longitude;  the  y-axis  shows  aircraft  altitude  (ft  above 
mean  sea  level);  b)  3-D  profile  of  the  same  flight.  The  translucent  “glass  sheet”  represents  the  cloud 
base.  Red  droplines  represent  IMC  penetration  during  this  flight;  yellow  droplines  represent  flight  in 
visual  meteorological  conditions  (VMC). 


Overall,  this  was  a  moderately  difficult,  dynamic 
weather  situation,  designed  to  gauge  how  pilots  would 
react  to  conditions  changing  from  easy  to  marginal  VFR. 
Therefore,  the  preflight  self-briefing  became  a  critical 
feature  of  this  mission.  The  better  that  briefing,  the  better 
the  “mental  model”  a  pilot  would  have  to  understand  both 
the  terrain  and  the  potential  weather  over  that  terrain. 

Emulation  of  aviationweather.gov.  Key  features  of  the 
popular  NWS  Web  site,  www.aviationweather.gov,  were 
emulated  by  the  author  in  Microsoft  Visual  Studio  2005  . 


Visual  Studio  is  a  programming  suite  that  enables  creation 
of  Hypertext  Markup  Language  (html) -based  Web  pages 
with  event-driven,  dynamic  content.  Such  emulated  Web 
pages  can  react  to  mouseovers,  mouse  clicks  and  releases, 
and  page-button  clicks  the  same  way  actual  Web  pages 
do.  With  the  addition  of  “code-behind,”8  a  program  can, 
for  instance,  display  flyout  menus,  popup  message  boxes, 
and  play  animated  .gif  movies.  Moreover,  code-behind 
can  also  record  both  what  is  acted  upon  onscreen  and 
when ,  making  it  a  useful  experimental  platform. 


8Code-behind  is  computer  code  that  “sits  behind”  the  .html  code 
that  created  the  actual  Web  page.  Within  that  html  can  lie  embedded 
instructions,  which  then  trigger  the  code-behind  to  execute  functions 
and  procedures  the  programmer  has  written.  For  instance,  opening 
or  closing  a  Web  page  can  record  the  date  and  time,  which  can  be 
used  to  calculate  pageview  duration ,  the  length  of  time  the  person 
had  that  page  open. 
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Figure  6.  a)  Home  page  of  the  real  aviationweather.gov  at  the  time  the  emulation  was 
created;  b)  The  experiment’s  opening  screen;  c)  Emulation  of  a  SIGMET/AIRMET. 
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Figure  6a  shows  the  home  page  of  the  actual  aviation- 
weather.gov  at  the  time  the  emulation  was  created.  Figure 
6b  shows  a  screenshot  of  the  experiment’s  opening  screen, 
used  to  record  a  pilot’s  identification  number  and  to  begin 
or  end  the  briefing.  Figure  6c  shows  the  emulation  of  a 
SIGMET/AIRMET,  brought  up  by  left-clicking  on  any 
pages  left  menu  bar,  moving  the  cursor  to  the  resulting 
flyout  menu,  and  then  releasing  the  left  mouse  button. 

Close  resemblance  between  the  real  aviationweather.gov 
and  our  emulation  was  enabled  by  downloading  sample 
NWS  graphics  and  core  html  code  (e.g.,  menus) — the 
computer  instructions  that  created  their  Web  pages.9 
Custom  graphics  of  our  scripted  weather  situation  were 
then  created  with  Canvas  (ACD,  2005)  and  enhanced 
by  adding  JavaScript  instructions  to  the  html  code,  as 
well  as  buttons  and  hotspots  responsive  to  Visual  Studio 
code-behind.  The  intent  was  not  to  exactly  duplicate 
aviationweather.gov  but  to  emulate  the  basic  look-and-feel 
and  functionality  of  its  menu  and  graphics. 

Information  content.  Appendix  A  shows  screenshots 
of  the  18  Web  pages  used  in  the  experiment.  Separate 
pages  were  created  for: 

1)  SIGMET^/AIRMET1 1 . . . (Java  tool)  graphical 

2)  Convective  SIGMETs  graphical 

3)  CCFP12... graphical  (looping  animated  .gif  movie) 

4)  Convective  outlook... graphical 
4.2  Categorical 

4.4  Tornado 
4.6  Hail 
4.8  Wind 

5)  Turbulence... graphical 

6)  Icing...  graphical 

7)  Winds/Temps...  text 


9Because  the  National  Weather  Service  is  a  U.S.  government  agency, 
its  html  code  and  graphics  are  public-domain. 

10Significant  Meteorological  Information 
1  Airman’s  Meteorological  Information 
12Collaborative  Convective  Forecast  Product 


8)  Prog  charts  graphical 

9)  TAF13...Java  tool  graphical,  with  popup  text 

10)  TAF  Station  model... graphical 

11)  FA14.. .text 

12)  PIREPs15...text  (not  used  in  this  experiment) 

13)  METARs16  (Java  tool)... graphical,  with  popup  text 

14)  Radar... graphical  (looping  NEXRAD17  animated 
.gif  movie) 

15)  Satellite... graphical  (looping  cloud  cover  animated 
.gif  movie) 

Animated  .gif  movies  of  NEXRAD  images  and  satellite 
clouds  were  created  by  downloading  individual  images 
from  weather  data  sites  {http://www.goes.  noaa.gov/srchwest. 
html  and  http://radar.weather.gov/ridge/RadarImg/NOR/). 
Separate,  time-stamped  frames  were  then  collated  in 
Canvas  to  create  movies  with  controlled  frame  rates. 

Dependent  variables.  Two  primary  dependent  variables 
were  examined  in  this  study: 

1.  Pageviews :  tallies  of  each  page  viewed  by  each  pilot 

2.  Pageview  duration:  length  of  time  each  page  was  open 
for  reading  by  each  pilot 

Total  briefing  time  for  each  pilot  was  then  generated 
as  the  sum  of  his  or  her  individual  page  view  durations. 

Since  this  was  only  a  preliminary  study,  sophisticated 
methods  such  as  eye  tracking  were  not  employed.  Such 
methods  would  be  the  next  logical  step,  however,  since 
knowing  what  page  has  been  opened  (pageviews),  and  for 
how  long  it  is  viewed  (pageview  duration),  are  necessary 
kinds  of  information — but  not  sufficient  to  fully  know 
exactly  what  on  a  given  page  was  viewed  or  what  infor¬ 
mation  was  understood.18 


13Terminal  Aerodrome  Forecast 
l4FA= Aviation  area  18-h  forecast 
15Pilot  Report 

16Meteorological  Aviation  Report 
17Next  Generation  Radar 

18Even  knowing  what  content  was  viewed  on  a  page  does  not  guarantee 
that  the  viewer  understood  what  was  viewed.  This  is  clearly  a  difficult 
problem.  Yet,  we  have  to  start  somewhere  if  we  want  to  understand 
and  improve  preflight  self-briefing. 
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Figure  7.  (Left).  Pageview  duration  frequency  distributions  for  the  part-task  emulation  of  aviationweather.gov.  (Left,  top)  Phase 
1  (N=50).  (Left,  bottom)  Phase  2  (N=44).  (Right,  bottom)  Pageview  durations  (medians  only).  (Right,  top)  Total  page  views.  The 
y-axis  represents  total  number  of  times  each  page  was  viewed  by  all  pilots  (Phase  2  values  adjusted  for  attrition). 


RESULTS 

Distributional  Normality  of  Data 

Pageviews  and  pageview  duration  both  produced 
frequency  distributions  (e.g.,  Figure  7,  left).  Ideally  such 
distributions  should  be  normal  (bell-shaped)  about  their 
means  to  justify  the  use  of  parametric  statistics. 

However,  in  both  Phases  1  and  2,  the  Web-emulation 
page  view  duration  data  were  distinctly  non-normal.  Some 
pages  were  open  for  very  little  time  while  a  few  were  open 
for  extraordinarily  long  times.  In  particular,  some  of  the 
longest  viewing  times  reflected  either  pilots  forgetting  to 
close  out  the  last  page  after  finishing  the  briefing  or  else 
moving  back  and  forth  between  the  computer  and  the 
sectional  during  the  briefing. 

Given  this  non-normality,  means  and  mean-based 
statistics  were  shunned  in  favor  of  rank-  and  mediant- 
based  (nonparametric)  statistics.  These  are  less  powerful, 
statistically,  but  more  robust  to  deviant  distributions  and 
data  outliers. 


Specific  Observations 

How  Web  preflight  briefing  changes  over  time.  Patterns 
of  page  usage  varied  from  Phase  1  to  Phase  2.  Figure  7 
(left)  compares  Phase  1  pageview  durations  (left,  top)  with 
Phase  2  (left,  bottom).  The  logarithmic  y-axis  makes  it 
easier  to  discriminate  small  values  while  still  represent¬ 
ing  large  values.  Box  plots  show  25th,  50th  (median), 
and  75th  percentiles.  By  comparing  groupwise  medians 
(rather  than  means),  we  expect  more  stable  pagewise 
viewing  estimates,  since  the  effect  of  outliers  is  reduced. 

Figure  7  (left)  shows  all  the  data  but  is  admittedly 
complex  and  confusing.  Therefore,  Figure  7  (right,  bot¬ 
tom)  depicts  the  same  data,  but  simplified,  showing  only 
groupwise  medians. 

These  data  form  two  patterns,  one  for  each  phase  of  the 
experiment.  Phase  1-2  correlation  can,  therefore,  estimate 
consistency-over-time  for  median  pageview  durations.20 


20These  medians  are  based  upon  pilots  who  actually  viewed  those 
pages  (as  opposed  to  the  alternate  technique  of  padding  distributions 
with  zeros  to  reflect  non- viewing  pilots). 


19The  median  is  the  mid-point  of  a  group  of  scores,  above  and  below 
which  half  the  score  values  lie. 
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Pageview  duration  showed  no  significant  consistency  over 
time  (pWilcoxon  =  •  14,  NS).21  Some  pages  were  viewed  longer 
in  Phase  2,  others  less. 

However .page preferences  (Total page  views,  Fig.  7,  right, 
top)  remained  remarkably  consistent  over  time  (rc 

£ x  J  v  Spearman 

.972,  p  <.000001).  In  other  words,  given  similar  weather 
situations,  pilots  tended  to  consistently  seek  out  certain 
kinds  of  information.  They  just  did  not  necessarily  spend 
consistent  amounts  of  time  viewing  that  information. 

Evidence  of  learning.  Next,  notice  the  significant  decrease 
in  page  views  from  Phase  1  to  Phase  2,  averaging  over  20%, 
P wiicoxon  =  -0002)  .22This  probably  does  not  mean  that  pilots 
started  disregarding  their  preflight  weather  briefings  in 
Phase  2.  It  probably  only  means  that  pilots  had  already 
used  the  computerized  system  once  and  were  showing 
increasing  skill  using  the  system  the  second  time  around. 

Information  accessible  from  the  top-level  menu  in 
one  operation  may  be  more  likely  to  be  accessed  than 
information  that  requires  multiple  operations.  This  is 
an  important  human  factors  hypothesis  that  should  be 
further  investigated.  Figure  7  (right,  top  and  bottom) 
shows  that  the  page  4  variants  (convective  outlook)  were 
low-view  and  low-duration  pages  in  both  Phases  1  and 
2 — except  for  the  “Categorical”  page  (page  “4.2”).  The 
Categorical  page  was  the  default,  and  the  other  three 
page  4  variants  had  to  be  accessed  by  way  of  buttons  on 
the  Categorical  page.  Hence,  this  indicates  that  most 
pilots  simply  never  pressed  those  buttons.  This  could 
imply  that  ease-of- access  is  a  critical  aspect  of  design,  or 
it  could  simply  mean  that,  given  this  particular  flight, 
the  information  on  those  particular  pages  was  judged  not 
particularly  useful  by  most  pilots. 

Page  popularity  versus  dijfculty.  In  a  sense,  some  pages 
did  seem  “more  popular”  than  others.  As  mentioned, 
the  page  4  variants  (Convective  Outlook)  uniformly 
received  very  few  views.  In  contrast,  pages  9  (TAF  Java 
Tool)  and  14  (Looping  NEXRAD)  received  the  highest 
number  of  views. 

Examining  this  issue  of  “popularity”  more  deeply, 
we  can  assume  that  people  tend  to  dwell  on  things  they 
find  either 

1 .  highly  informative  or 

2.  hard  to  understand 


21This  was  partly  due  to  huge  decreases  in  Phase  2  median  view 
duration  for  two  outliers — pages  7  and  1 1 — the  only  two  text-based 
pages  (all  others  were  graphical).  Page  7  was  winds/temps  as  text; 
page  1 1  was  the  area  forecast  (FA)  as  text.  From  this,  we  might  be 
tempted  to  argue  that  text-based  products  are  less  efficient  than 
graphical  products,  but  that  would  not  take  into  account  differences 
in  the  amount  of  information  per  page.  Plus,  when  pages  7  and  1 1 
are  ignored,  rSpearman  ®  -399 — less  of  a  change  than  we  might  expect. 

22The  statistical  significance  was  not  just  trivially  due  to  the  loss  of  6 
pilots  in  Phase  2.  The  vertical  (y)  axis  of  Figure  3  (right)  is  corrected 
for  attrition  (Phase  2  page  views  were  multiplied  by  30/44). 


Unfortunately,  explanation  1  implies  something  good, 
explanation  2  not  so  good.  Moreover,  either  explanation — 
or  both — may  be  confounded  in  any  given  pilot  for  any 
given  page.  This  confound  will  have  to  be  untangled  in 
future  studies  using  more  sophisticated  methods. 

For  now,  let  us  just  imagine  a  simple  “group  dwell 
index,”  calculated  by  multiplying  each  pages  total  group 
number  of  page  views  times  its  median  view  duration. 
Dwell  can  then  estimate  “group  attention  paid  to  each 
page,”  which  can  serve  as  a  temporary  proxy  for  either 
“popularity”  or  “difficulty,”  to  be  determined  at  a  later  date. 

Dwell  =  Ni*Ti  (1) 

where  N  =  the  ith  pages  number  of  views  andT  =  view 
duration  (^  above  a  variable  is  the  symbol  for  “median”). 
Figure  8  illustrates.  Figure  8a  may  or  may  not  support 
the  point  made  previously  about  the  importance  of  page 
design  (this  needs  further  investigation) .  The  three  page-4 
non-default  variants  (that  took  more  effort  to  access)  had 
far  lower  dwell.  As  previously  mentioned,  these  required  a 
button  click,  and  many  pilots  simply  ignored  the  buttons. 
Figure  8  b  shows  clearly  how  pages  tended  to  cluster  into 
two  groups,  high-  versus  low-dwell. 

Table  2  shows  Phase  2  pages  ranked  by  dwell.  Of 
those,  Figure  9  illustrates  the  three  highest-ranked  pages. 

METARs  (page  13)  andTAFs  (page  9)  lost  dwell  in 
Phase  2.  That  decrease  probably  reflected  increased  pilot 
skill  in  using  the  mouseover  popup  text  on  those  two  pages, 
which  may  have  been  initially  confusing.  Nonetheless, 
METARs  and  TAFs  still  remained  above  the  median  in 
dwell  in  both  phases. 

Effect  ofWeb  preflight  briefing  time  on  subsequent  flight 
safety.  One  could  argue  that  briefing  is  as  briefing  does. 
This  means  we  should  try  to  assess  what  effect  the  qual¬ 
ity  of  a  preflight  briefing  has  on  subsequent  flight  safety. 

The  results  of  this  study  imply  that  the  mere  quantity 
of  preflight  weather  self-briefing  is  not  synonymous  with 
its  quality.  Total  briefing  time  ( Web  preflight  duration)  may 
be  the  simplest  candidate  for  a  metric  of  “quality.”  Yet,  the 
sheer  amount  of  time  pilots  spent  on  preflight  weather 
briefing  did  not  seem  to  influence  their  safety  record  later 
during  the  flight.  Correlations  ofWeb  preflight  duration 
with  safety  variables  such  as  flight  duration  (here,  a  proxy 
for  degree  of  penetration  into  deteriorating  weather), 
minimum  distance  to  ABQ,  minutes  scud  running, 
minutes  in  IMC,  and  minutes  <  500’  AGL,  were  all  non¬ 
significant,  with  r-values  ranging  from  -.076  <  r  <.109. 

Durable  relations  between  variables.  There  were  two 
aspects  of  the  briefing  that  did  relate  to  pilot  demographic 
variables.  If  we  define  a  “durable”  relation  as  one  remain¬ 
ing  statistically  significant  across  both  Phases  1  and  2, 
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Dwell,  Phase  2 
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Figure  8.  a)  Page  dwell  (Equation  1);  b)  Log-log  scatterplot  of  Phase  1  by  Phase  2  dwell,  with  power  function 
best-fit  line.  Pages  below  the  dashed  identity  line  (e.g.,  8,  2,  15)  gained  dwell  in  Phase  2.  Again,  all  Phase  2 
values  were  adjusted  for  pilot  attrition. 


Figure  9.  Phase  2’s  three  highest-dwell  pages. 
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Table  3.  Durable,  non-trivial  relations  between  Phases  1  and  2  variables. 

Variable  2 
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Web  Preflight  Duration 


-.348  (p=.013)  Phi 
-.421  (p=  004)  Ph2 


.417  (p=.003)  Phi 
.475  (p=.001)  Ph2 


rPb  =  Point-biserial.  rs  =  Spearman  rho.  P-values  are  in  parentheses. 


then  Table  3  shows  the  two  durable  correlations  observed 
between  preflight  briefing  and  pilot  demographics. 

Table  3  shows  that  both  local  pilots  and  younger  pilots 
spent  slightly  less  time  on  their  Web  weather  preflight 
briefing.  However,  this  is  not  surprising.  Local  pilots  were 
likely  just  more  familiar  with  local  terrain  and  weather 
patterns.  And,  older  pilots  may  have  been  either  slightly 
more  careful  briefers,  or  they  might  have  simply  been  a 
bit  less  familiar  with  Web-based  briefing  (or  with  avia- 
tionweather.gov  itself). 

If  there  is  a  message  in  these  durable  relations,  it  is 
that  preflight  briefing  systems  need  to  be  designed  to 
accommodate  not  just  the  technologically  sophisticated 
user  but  also  the  less  sophisticated  user. 

DISCUSSION 

Motivation  for  This  Study 

The  Internet  has  become  increasingly  capable,  af¬ 
fordable,  and  popular.  Internet-based  preflight  weather 
self-briefing  is  following  suit.  Therefore,  the  motivation 
for  this  research  was  to  begin  systematic  investigation  of 
how  modern  Web-based  weather  products  are  being  used 
by  GA  pilots  during  this  kind  of  briefing. 

Method 

An  experimental,  data-gathering  emulation  of  the 
National  Weather  Service  Web  site  www.aviationweather. 
gov  was  written  by  the  author  for  use  on  a  standalone 
PC.  This  was  tested  for  use  with  2  similar,  challenging 
weather  scenarios  involving  50  GA  pilots  in  simulated 
cross-country  VFR  flight.  Each  pilot  made  a  total  of  2 
flights,  separated  by  several  months.  Each  flight  employed 
1  of  2  similar  weather  scenarios,  with  the  Web  emulation 
used  as  the  pilots5  sole  preflight  weather  briefing  source. 
Eighteen  Web  pages  were  created  to  display  weather 
information  for  each  scenario,  each  page  emulating  a 
similar  category  of  product  shown  by  aviationweather.gov . 

Measurements.  Two  dependent  variables  were  mea¬ 
sured — page  views  (which  pages  each  pilot  viewed)  and 
pageview  duration  (how  long  each  page  was  viewed). 
Total  briefing  time  was  then  calculated  for  each  pilot  by 


summing  pageview  durations.  A  groupwise  “dwell55  index 
was  also  derived  for  each  Web  page  by  multiplying  the 
groups  page  views  times  its  median  pageview  durations. 
Dwell  then  could  serve  as  a  proxy  for  either  “popularity55 
or  “difficulty”  (see  below). 

Methodological  difficulties  discovered.  Three  main 
methodological  difficulties  were  discovered.  First,  a  small- 
but-significant  number  of  pilots  failed  to  close  out  their 
briefing  session  after  finishing,  and/or  divided  their  time 
between  the  computer  and  the  sectional.  Both  cases  led 
to  the  same  problem — that  recorded  pageview  durations 
occasionally  did  not  reflect  actual  viewing  times,  leading 
to  data  non-normality.  Fortunately,  this  problem  was 
minimized  by  the  use  of  median-based  statistics,  which 
are  more  robust  to  data  non-normality  and  outliers. 

Second,  pageview  durations  and  total  briefing  time 
are  not  necessarily  synonymous  with  quality  of  brief¬ 
ing.  “Quality55  will  eventually  need  to  be  assessed  more 
directly  by  other  methods:  for  instance,  by  eye- tracking 
and  by  post-briefing  knowledge  tests.  Quality  can  also 
be  indirectly  assessed  (as  was  done  here)  by  correlating 
aspects  of  the  preflight  briefing  with  objective  behavioral 
measures  of  subsequent  flight  performance  (e.g.,  terrain 
and  ground  clearance,  penetration  distance  into  decreas¬ 
ing  visibility,  and  so  forth). 

Third,  longer  pageview  durations  do  not  necessarily 
mean  that  a  given  page  is  highly  informative.  It  can  sim¬ 
ply  mean  that  the  page  is  difficult  to  understand.  Again, 
more  sophisticated  methods  will  be  required  to  sort  out 
this  confound. 

Main  Findings 

First  and  foremost,  this  Web  emulation  was  successfully 
used  by  these  pilots  as  a  standalone  preflight  weather  self¬ 
briefing  source.  As  an  experimental  platform,  it  enabled 
objective  performance  data-capture  in  tab-delimited 
text  file  formats  exportable  to  spreadsheets  and  statisti¬ 
cal  programs.  As  far  as  we  know,  this  is  the  first  such 
experimental  platform  devised  for  the  express  purpose 
of  testing  Web-based  preflight  weather  briefing. 

After  data  analysis,  a  number  of  specific  findings 
emerged.  First,  given  a  specific  flight  mission,  we  found 
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that,  as  a  group,  GA  pilots  seemed  significantly  consistent 
over  time  in  the  kinds  of  information  they  sought  out. 
However,  the  amount  of  time  they  spent  examining  that 
information  seemed  to  vary  over  time,  even  given  similar 
flight  situations. 

Second,  pilots  seem  to  have  favorite  information  sources 
(e.g.,  graphical  prog  charts,  TAFs,  NEXRAD,  and  satellite 
cloud  images).  Yet,  we  should  expect  those  favorites  to 
shift  somewhat,  depending  on  the  flight  mission.  Since 
only  one  basic  type  of  mission  was  tested  here,  further 
research  is  necessary  on  this  topic. 

Third,  Web  page  design  is  potentially  important.  For 
example,  pages  not  accessible  from  the  top-level  menu 
may  be  ignored  or  overlooked. 

These  findings  are  important  to  weather  information 
providers  because  the  ability  to  display  certain  types  of 
information — and  certain  ways  of  displaying  it — are  sure 
to  give  the  knowledgeable  provider  an  advantage  over 
competitors.  The  key  will  be  highly  informative  pages 
that  still  remain  easy  to  understand. 

Secondary  findings.  As  expected,  skill  at  using  the  system 
appears  to  improve  with  practice. 

There  were  also  slight-but-significant  tendencies  for 
older  pilots  and  out-of-town  pilots  to  spend  a  bit  more 
time  on  their  preflight  briefing.  Older  pilots  probably 
have  slightly  lower  average  computer  skills  and  may  be 
more  used  to  getting  briefed  by  the  Flight  Service  Station. 
Additionally,  the  greater  briefing  times  of  out-of-town 
pilots  were  probably  merely  due  to  their  being  less  familiar 
with  the  local  geography  and  weather  patterns. 


Future  Directions 

During  this  experiment,  many  pilots  informally  ex¬ 
pressed  the  opinion  that  the  future  of  weather  briefing 
looks  increasingly  Internet-based,  as  opposed  to  coming 
solely  from  the  Flight  Service  Station.  The  rapid  devel¬ 
opment  of  Internet  technology  certainly  supports  these 
opinions. 

If  the  future  truly  is  Web-based,  then  pilot  training  will 
need  to  address  these  new  technologies  and  trends.  Pilots 
will  need  to  become  skilled  at  self-briefing  through  prod¬ 
ucts  such  as  aviationweather.gov.  Hence,  these  products 
themselves  should  become  the  subject  of  human  factors 
research  through  such  methods  as  content  analysis  and 
usability  testing. 

More  specifically,  we  suspect  that  graphical  display  of 
information  is  superior  to  text  display.  This  is  an  empirical 
issue  that  should  be  tested,  since  the  current  study  does 
not  directly  address  that  hypothesis. 

Similarly,  other  aspects  of  page  design  should  be 
explored.  A  series  of  usability  tests  could  be  devised  to 
maximize  information  relevance  and  optimize  ease-of-use. 

Metrics  of  briefing  quality  are  needed.  Page  views  and 
pageview  duration  are  interesting — but  not  definitive — 
measures  of  what  a  pilot  gets  out  of  a  briefing.  Eye  tracking 
could  allow  a  cross-check  of  how  much  attention  is  being 
paid  to  an  overall  page,  as  well  as  to  specific  parts  of  that 
page.  Short  questions  could  also  be  created  to  test  pilot 
situational  knowledge.  Finally,  these  measures  could  be 
correlated  with  objective,  behavioral  flight  safety  measures 
to  test  if  “briefing  is  as  briefing  does.” 

Other  testable  ideas  are  user-configurable  displays  (ones 
capable  of  giving  the  user  a  choice  of  how  information 
will  be  displayed)  and  4D  route-planning  displays23 — ones 
that  give  a  sense  of  what  weather  may  be  encountered 
where  and  when  along  a  selected  route,  given  the  aircraft 
speed,  planned  altitude,  and  direction  of  travel  of  both 
aircraft  and  weather  systems. 


23Route-planning  displays,  such  as  the  Flight  Path  Tool,  are  currently 
being  developed  and  fielded  by  the  Aviation  Digital  Data  Service 
(ADDS,  http://weather.aero/) — a  partnership  between  NOAA’s 
Forecast  Systems  Laboratory,  the  National  Center  for  Atmospheric 
Research  (NCAR)  Research  Applications  Program,  and  the  National 
Centers  for  Environmental  Prediction  (NCEP)  Aviation  Weather 
Center. 
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APPENDIX  A 

Web  Preflight  Briefing  Screenshots 
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